Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization

Abstract In this paper, we present a limited-memory common-directions method for smooth optimization that interpolates between first- and second-order methods. At each iteration, subspace of limited dimension size is constructed using first-order information from previous iterations, an efficient Newton deployed to find approximate minimizer within subspace. With properly selected as small two, the proposed algorithm achieves optimal convergence rates methods while remaining descent method, it also possesses fast speed on nonconvex problems. Since major operations our are dense matrix-matrix operations, can be efficiently parallelized in multicore environments even sparse By wisely utilizing historical information, communication-efficient distributed uses multiple machines steps calculated with little communication. Numerical study shows has superior empirical performance real-world large-scale machine learning